Mirroring in image guided surgery

ABSTRACT

An imaging system, including a head-mounted display worn by a system operator. A marker defines a plane when attached to a human subject. Optically reflective elements are disposed on the marker and on opposing sides of the plane in a non-symmetrical arrangement with respect to the plane. A memory stores a graphical representation of a tool used in a procedure performed on the human subject, and an image of anatomy of the human subject. A camera attached to the display acquires an image of the marker and the tool. A processor analyzes the image to identify the plane and to identify a side of the plane wherein the camera is located, and to render to the display the image of the anatomy of the human subject with the graphical representation of the tool superimposed thereon from a point of view in the identified side of the plane.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/724,297, filed Dec. 22, 2019, which is incorporated herein byreference.

FIELD OF THE INVENTION

This invention relates generally to an augmented reality system, andspecifically to correct image projection when it is used in image guidedsurgery.

BACKGROUND

Correct imaging is important in image guided surgery, and a number ofsystems are known in the art for producing correct imaging.

U.S. Pat. Nos. 7,630,753 and 9,757,087, to Simon et al., describe asurgical instrument navigation system that allows a surgeon to invertthe three-dimensional perspective of the instrument to match theirperspective of the actual instrument.

U.S. Pat. No. 9,538,962, to Hannaford et al., describes a system forproviding networked communications. The system includes a plurality ofhead-mountable devices, each in communication with a control system viaa communication network.

U.S. Pat. No. 9,710,968, to Dillavou et al., describes a system for roledesignation with multiple sources.

U.S. Pat. No. 9,886,552, to Dillavou et al., describes a method forimage registration that includes rendering a common field of interestthat reflects a presence of a plurality of elements. At least one of theelements is a remote element located remotely from another of theelements.

U.S. Pat. No. 9,940,750, to Dillavou et al., describes a method for rolenegotiation that can comprise rendering a common field of interest thatreflects a presence of a plurality of elements. At least one of theelements is a remote element located remotely from another of theelements.

U.S. Pat. No. 9,959,629, to Dillavou et al., describes a method formanaging spatiotemporal uncertainty in image processing. The method cancomprise determining motion from a first image to a second image.

U.S. Pat. No. 10,194,131, to Casas, describes a real-time surgery methodfor displaying a stereoscopic augmented view of a patient from a staticor dynamic viewpoint of the surgeon. The method employs real-timethree-dimensional surface reconstruction for preoperative andintraoperative image registration.

US Patent Application 2011/0216060, to Weising et al., describes amethod for controlling a view of a virtual scene with a portable device.A signal is received and the portable device is synchronized to make thelocation of the portable device a reference point in a three-dimensional(3D) space.

US Patent Application 2017/0027650, to Merck et al., describes receivingdata characterizing a mother video feed acquired by an endoscopic videocapture device. The mother video feed can be for characterizing anoperative field within a patient.

US Patent Application 2017/0251900, to Hansen et al., describes adepiction system for generating a real time correlated depiction ofmovements of a surgical tool for uses in minimally invasive surgery.

US Patent Application 2017/0367771, to Tako et al., describes a virtualreality surgical navigation method that includes a step of receivingdata indicative of a surgeon's current head position, including adirection of view and angle of view of the surgeon.

US Patent Application 2018/0247128, to Alvi et al., describes a systemfor accessing a surgical dataset including surgical data collectedduring performance of a surgical procedure. The surgical data caninclude video data of the surgical procedure.

Documents incorporated by reference in the present patent applicationare to be considered an integral part of the application except that, tothe extent that any terms are defined in these incorporated documents ina manner that conflicts with definitions made explicitly or implicitlyin the present specification, only the definitions in the presentspecification should be considered.

SUMMARY

An embodiment of the present invention provides an imaging system,consisting of:

a head-mounted display configured to be worn by an operator of thesystem;

a marker configured to be attached to a human subject and defining aplane when attached to the human subject, the marker having opticallyreflective elements disposed on the marker and on opposing sides of theplane in a non-symmetrical arrangement with respect to the plane;

a memory configured to store a graphical representation of a tool usedin a procedure performed by the operator on the human subject, and animage of anatomy of the human subject;

a camera attached to the display and configured to acquire an inputimage of the marker and of the tool; and

a processor configured to analyze the input image so as to identify theplane and to identify a side of the plane wherein the camera is located,and to render to the display the image of the anatomy of the humansubject with the graphical representation of the tool superimposedthereon from a point of view in the identified side of the plane.

In a disclosed embodiment the plane makes an angle between +20° and −20°with a sagittal plane of the human subject. Alternatively, the planemakes an angle between +20° and −20° with an axial plane of the humansubject.

In a further disclosed embodiment the marker has a two-dimensionalsurface which makes an angle between +20° and −20° with a frontal planeof the human subject.

In a yet further disclosed embodiment the marker defines a further planeand the optically reflective elements are disposed on opposing sides ofthe further plane in a non-symmetrical arrangement with respect to thefurther plane, and the processor is configured to analyze the inputimage so as to identify the further plane and to identify a side of thefurther plane wherein the camera is located, and to render to thedisplay the image of the anatomy of the human subject with the graphicalrepresentation of the tool superimposed thereon from a point of view inthe identified side of the further plane. Typically, the plane and thefurther plane are orthogonal to each other.

In an alternative embodiment the camera is located at a vertical heightabove the marker, and the processor is configured:

to ascertain the vertical height in response to the acquired input imageof the marker;

to calculate a pair of planes, each of the pair having a preset acuteangle to the identified plane and defining a first acute-angled wedgeregion and a second acute-angled wedge region to the identified plane;and

when the display moves so that the point of view crosses the firstacute-angled wedge region and the second acute-angled wedge region, orbegins within the first acute-angled wedge region and crosses the secondacute-angled wedge region, while the camera remains at the verticalheight, to render to the display the image of the anatomy of the humansubject with the graphical representation of the tool superimposedthereon from the point of view of a region opposite the identified side.

Typically the preset acute angle is less than or equal to 10°.

In a further alternative embodiment the camera is located at a verticalheight above the marker, and the processor is configured:

to ascertain the vertical height in response to the acquired input imageof the marker; and

when the display moves so that the vertical height changes, to renderunchanged to the display the image of the anatomy of the human subjectwith the graphical representation of the tool superimposed thereon.

There is further provided, according to an embodiment of the presentinvention, an imaging system, consisting of:

a first head-mounted display configured to be worn by a first operatorof the system;

a second head-mounted display configured to be worn by a second operatorof the system;

a marker configured to be attached to a human subject and defining aplane when attached to the human subject, the marker having opticallyreflective elements disposed on the marker and on opposing sides of theplane in a non-symmetrical arrangement with respect to the plane;

a memory configured to store a graphical representation of a tool usedin a procedure performed by the first operator on the human subject, andan image of anatomy of the human subject;

a first camera attached to the first display and configured to acquire afirst input image of the marker and of the tool;

a second camera attached to the second display and configured to acquirea second input image of the marker and of the tool; and

a processor configured to:

analyze the first input image so as to identify the plane and toidentify a first side of the plane wherein the first camera is located,and to render to the first display the image of the anatomy of the humansubject with the graphical representation of the tool superimposedthereon from a first point of view in the identified first side of theplane, and

analyze the second input image so as to identify the plane and toidentify a second side of the plane wherein the second camera islocated, and to render to the second display the image of the anatomy ofthe human subject with the graphical representation of the toolsuperimposed thereon from a second point of view in the identifiedsecond side of the plane.

There is further provided, according to an embodiment of the presentinvention, a method, consisting of:

providing a head-mounted display configured to be worn by an operator ofan imaging system;

attaching a marker to a human subject, the marker defining a plane whenattached, the marker having optically reflective elements disposed onthe marker and on opposing sides of the plane in a non-symmetricalarrangement with respect to the plane;

storing in a memory a graphical representation of a tool used in aprocedure performed by the operator on the human subject, and storing animage of anatomy of the human subject in the memory;

attaching a camera to the display;

acquiring an input image of the marker and of the tool with the camera;and

analyzing the input image so as to identify the plane and to identify aside of the plane wherein the camera is located, and to render to thedisplay the image of the anatomy of the human subject with the graphicalrepresentation of the tool superimposed thereon from a point of view inthe identified side of the plane.

The present disclosure will be more fully understood from the followingdetailed description of the embodiments thereof, taken together with thedrawings. A brief description of the drawings follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of an initial preparatory stage of amedical procedure, according to an embodiment of the present invention;

FIGS. 2, 3, and 4 are schematic depictions of entities used in theinitial stage, according to an embodiment of the present invention;

FIG. 5 is a flowchart of steps performed to register a patient markerwith the anatomy of a patient during the initial preparatory stage;

FIG. 6 is a schematic illustration of a subsequent stage of theprocedure, according to an embodiment of the present invention;

FIG. 7 is a flowchart of steps performed during the subsequent stage,according to an embodiment of the present invention;

FIG. 8 shows schematic figures illustrating images generated in thesubsequent stage, according to an embodiment of the present invention;

FIG. 9 is a schematic top-down view of a surface of a marker used in theprocedure; and

FIG. 10 is a schematic illustration of the subsequent stage of theprocedure when there are two operators for the procedure, according toan embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS Overview

A head-mounted display, for a medical procedure that implements animaging system, such as an augmented reality system, in the display,typically needs to access stored computerized tomography (CT) files ofthe anatomy of a human subject. The display is worn by an operator ofthe system, and the accessed files are presented to the operator asscanned planes of the subject in the display. However, for thepresentation to be correctly oriented, it is necessary to know theposition of the operator with respect to the subject.

Embodiments of the present invention provide an imaging system thatdetermines the operator position automatically, and so displays an imageof the patient anatomy, and of a tool used in the procedure,automatically.

In addition to a head-mounted display (HMD) that is worn by an operatorof the system, the system comprises a marker that is attached to thehuman subject. The marker defines a plane of asymmetry when attached tothe human subject, since the marker has optically reflective elementsdisposed on the marker and on opposing sides of the plane in anon-symmetrical arrangement with respect to the plane. The plane ofasymmetry is typically approximately parallel to one of the mainanatomical planes of the human subject.

In the imaging system a memory stores a graphical representation of atool used in the procedure performed by the operator, and the memoryalso stores an image of the anatomy of the human subject. A camera isattached to the HMD, and acquires an input image of the marker and ofthe tool. A processor analyzes the input image so as to identify theplane and to identify a side of the plane wherein the camera is located.The processor then renders to the display the image of the anatomy ofthe human subject with the graphical representation of the toolsuperimposed thereon from a point of view in the identified side of theplane.

Detailed Description

In the following, all directional references (e.g., upper, lower,upward, downward, left, right, top, bottom, above, below, vertical, andhorizontal) are only used for identification purposes to aid thereader's understanding of the present invention, and do not createlimitations, particularly as to the position, orientation, or use ofembodiments of the invention.

In the description, like elements in the drawings are identified by likenumerals, and like elements are differentiated as necessary by appendinga letter to the identifying numeral.

Reference is now made to FIGS. 1, 2, 3, and 4 , which are diagramsaccording to an embodiment of the present invention. FIG. 1 is aschematic illustration of an initial preparatory stage of a medicalprocedure using an imaging system 20, and FIGS. 2, 3, and 4 areschematic depictions of entities used in the initial stage. The medicalprocedure exemplified here is performed on the back of a human subject22, herein also termed patient 22, and during the initial stage of theprocedure an operator 26 of system 20, also herein termed medicalprofessional 26 makes an incision 24 into the patient's back. Theprofessional inserts a spinous process clamp 30 into the incision, sothat opposing jaws of the clamp are located on opposite sides of thespinous processes. The professional then slides the clamp over thevertebral laminas, and adjusts the clamp to grip one or more spinousprocesses, selected by the professional, of the patient. Clamp 30 isdescribed below with reference to FIG. 4 , and a clamp such as clamp 30is described in more detail in U.S. Patent Application 2019/0175228which is incorporated herein by reference.

Clamp 30 acts as a support for a patient marker 38, which is attachedrigidly to the clamp. During substantially all of the procedure, i.e.,during the initial, as well as the subsequent stages, patient marker 38is used as a fiducial for patient 30, since because of its rigidconnection to the patient, any movement of the patient is reflected in acorresponding motion of the patient marker. In order to operate as sucha fiducial, in embodiments of the present invention, in the initialstage of the procedure marker 38 is registered with the anatomy ofpatient 30, herein assumed to comprise the skeleton of the patient, asis described herein.

During the procedure medical professional 26 wears a head-mounteddisplay (HMD) 64 which is configured to present stored images, that arealigned with patient 22, to professional 26. HMD 64 is described furtherbelow.

As is also described below, in serving as a fiducial, marker 38 performstwo functions: a first function wherein the marker is used to maintainregistration between frames of reference of the head-mounted display andthe patient's anatomy, and a second function wherein the marker is usedto ascertain where the medical professional is located with respect tothe patient. Thus, for the second function, the marker provides alocation of the medical professional as being on a left side or a rightside of the patient, or on an upper side or a lower side of the patient.

An augmented reality head-mounted display such as HMD 64 is described inmore detail in U.S. Patent Application 2017/0178375 which isincorporated herein by reference.

During the initial stage of the procedure, a registration marker 40 isplaced on the patient's back, and is used to implement the registrationof patient marker 38 with the anatomy of patient 30. In contrast topatient marker 38, registration marker 40 is typically only used duringthe initial stage of the procedure, i.e., for the registration of thepatient marker 38, and once the registration has been performed, for thesubsequent procedure stages the registration marker may be removed fromthe patient's back. As will be apparent from the following description,only registration marker 40 is subject to fluoroscopy, and patientmarker 38 is not subject to fluoroscopy.

Also during the initial stage of the procedure, a camera 42, fixedlyattached to head-mounted display 64, is used to image the registrationmarker and the patient marker. Camera 42 typically operates in thevisible and/or near-visible spectrum, i.e., at wavelengths ofapproximately 300 nm-900 nm.

A processing system 28 is coupled, by cables and/or wirelessly, tocamera 42. System 28 comprises a computer processor 32, a memory 33comprising stored images 35 that include images 304, 308, and 324,described below, a screen 34, and an input device 36 such as a pointingdevice. The system is configured to analyze the images acquired by thecamera, as is described further below. Other functions of system 28 arealso described below.

In order to operate, HMD 64 is coupled to processor 32 of system 28, oralternatively HMD 64 has its own dedicated processor which performssimilar functions to those performed by processor 32. When HMD 64 isoperative it presents stored images, that are aligned with patient 22,to professional 26.

FIGS. 2 and 3 are respectively schematic perspective and cross-sectionalviews of registration marker 40, which is assumed to define aregistration marker frame of reference 50, herein assumed to comprise anorthogonal set of xyz axes. Marker 40 is formed from a solid substrate44, which is opaque to light in the visible and near-visible spectrum,and which is transparent to fluoroscopic radiation. Substrate 44 istypically formed from a hard plastic, such as polycarbonate, but anyother solid material which is opaque to light and transparent tofluoroscopic radiation may be used in embodiments of the presentinvention.

In the illustrated embodiment of marker 40, substrate 44 is formed as arectangular parallelepiped 46, upon which is mounted a pillar 48.

A plurality of optically reflective, but radiotransparent, discreteelements 54 are disposed on substrate 44. Elements 54 are hereinbelow,by way of example, assumed to comprise discs, and are also referred toherein as discs 54. It is understood that said optically reflective andradiotransparent elements may be of different shapes and/or sizes.

Some of the plurality of discs 54 are fixedly attached, typically bycementing, to a two-dimensional (2D) surface 52 of parallelepiped 46.These discs 54 are formed in a generally rectangular 2D pattern onsurface 52. In addition, an optically reflective disc 54 is alsocemented onto pillar 48, so that there is in totality athree-dimensional (3D) array of discs 54 disposed on the substrate. The3D array of discs 54 are distributed on 2D surface 52, and on pillar 48,so that when marker 40 is illuminated and imaged by camera 50 the discsare easily distinguished from substrate 44. Furthermore, as explained inmore detail below, the arrangement of discs 54 are configured to enableprocessor 32 to unambiguously determine the orientation and position offrame of reference 50 from the marker image.

The distributed discs 54 are herein assumed to comprise an opticalcomponent 56 of marker 40 that forms an optical pattern 58 for themarker. In a particular aspect of the invention optical pattern 58,comprising the distribution of discs 54, is implemented so that thepattern has no axis of symmetry and no plane of symmetry. The absence ofboth an axis and a plane of symmetry in the pattern ensures that theunambiguous determination of the orientation and position of the frameof reference of marker 40 is possible from the marker image for multipledifferent orientations and positions of the marker, the positions beingtypically within a region approximately 20 cm from the patient marker.

The description above of optical pattern 58 assumes that discs 54 areconfigured in three dimensions. However, as long as the pattern has noaxis of symmetry and no plane of symmetry, the discs forming the patternmay be arranged in only two dimensions, for example, absent the disc onpillar 48. Thus, pattern 58 may be formed in at least two dimensions,i.e., in the case of discs 54, as a two-dimensional array of the discsor as a three-dimensional array of the discs.

It will be understood that the requirement for discs 54 to be arrangedto form a pattern having an absence of both an axis and a plane ofsymmetry may be achieved using discs of substantially the same size andshape, wherein locations of the discs are selected so that the locationsare arranged to have the absence of both an axis and a plane ofsymmetry. The described pattern is hereinbelow referred to as a uniqueoptical pattern.

Alternatively, the unique optical pattern may be achieved using discs ofdifferent sizes and/or shapes. In this case, the locations of the discsmay also satisfy the requirement, but this is not a necessity.

A multiplicity of radiopaque elements 60 are disposed in substrate 44 bybeing embedded in a distribution within parallelepiped 46. Thedistribution of elements 60 is arranged in a two dimensional radiopaquepattern 62 such that, as for the pattern of discs 54, the radiopaquepattern has no axis of symmetry and no plane of symmetry. Becausesubstrate 44 is radiotransparent, and because of the absence of both anaxis and a plane of symmetry in radiopaque pattern 62, a fluoroscopic,typically computerized tomography (CT), scan of the radiopaque elementsof marker 40 enables the orientation and position of frame of reference50 to be unambiguously determined by processor 32 from the fluoroscopicscan. In one embodiment elements 60 comprise spheres which aredistributed in a 2D generally rectangular 2D pattern that issubstantially the same as the rectangular pattern of discs 54 on surface52.

The description above of elements 60 assumes that they are arranged in aradiopaque pattern of two dimensions. However, as long as the patternhas no axis of symmetry and no plane of symmetry, the elements formingthe pattern may also be arranged in three dimensions, for example, byincorporation of a radiopaque element 60A, substantially similar toelements 60, in pillar 48. Thus, pattern 62 may also be formed in atleast two dimensions, i.e., in the case of elements 60 and 60A, as atwo-dimensional array of elements 60 or as a three-dimensional array ofelements 60 and 60A.

As for discs 54, it will be understood that the requirement for elements60 to be arranged to form a pattern having an absence of both an axisand a plane of symmetry may be achieved using elements of substantiallythe same size and shape, wherein locations of the elements are selectedso that the locations are arranged to have the absence of both an axisand a plane of symmetry. The described pattern is hereinbelow referredto as a unique radiopaque pattern.

Alternatively, the unique radiopaque pattern may be achieved usingelements of different sizes and/or shapes. In this case, the locationsof the elements may also satisfy the requirement, but this is not anecessity.

The X-ray wavelengths of the CT scan are assumed to be in a range of0.01-10 nm.

The above description of marker 40 assumes that discs 54 and elements 60have different functionalities—the discs being optically reflective andradiotransparent, and the elements being radiopaque. In an alternativeembodiment of marker 40 at least some of discs 54 are configured to havedual functionality by being optically reflective and radiopaque. As forthe embodiment described above, in the alternative embodiment discs 54are configured and distributed on substrate 44 so that an optical imageof marker 40 provides an unambiguous determination of the orientationand position of frame of reference 50, and a fluoroscopic scan of themarker also provides an unambiguous determination of the orientation andposition of the frame of reference.

The physical construction of the illustrated embodiment of marker 40, asa pillar attached to a rectangular parallelepiped, comprising an arrayof discs 54 and an array of elements 60, is but one example of possiblephysical constructions of the marker that enables an unambiguousdetermination of the marker's position and orientation from a cameraimage and from a fluoroscopic scan. In a disclosed embodiment, ratherthan marker 40 comprising pillar 48 mounted on substrate 44, anindentation (in place of the pillar) is formed within the substrate, anda disc 54 is located on a surface of the indentation.

Other suitable constructions for marker 40 are also considered to bewithin the scope of the present invention.

For example, the substrate of marker 40, rather than being formed from aparallelepiped with a pillar or an indentation, may be formed assubstantially any conveniently shaped solid object that is opaque tolight in the visible and near-visible spectrum and which is transparentto fluoroscopic radiation.

In addition, rather than the optical component of marker 40 beingcomprised of a plurality of discs 54 arranged in a particular pattern,the component may comprise any array or pattern of optical elements thatis attached to the substrate, that is diffusely and/or specularlyreflective, and that is configured to have the absence of axes andplanes of symmetry described above, so that when imaged in visible ornear-visible light an unambiguous determination of the marker's positionand orientation may be made.

Referring to FIG. 4 , patient marker 38 is assumed to define a patientmarker frame of reference 100, assumed to comprise an orthogonal set ofxyz axes. In the embodiment illustrated in FIG. 4 marker 38 comprises arectangular parallelepiped substrate 102 to which is attached a tongue104 used to fixedly connect the substrate to clamp 30. A center 103 ofan upper surface of substrate 102 acts as an origin of the xyz axes.

The connection to clamp 30 is by a removable screw 112, and the patientmarker connects in a predetermined fixed spatial relationship to theclamp using holes 114 which align with studs 116 of the clamp. Substrate102 comprises a solid opaque material, and may be formed from anyconvenient material such as polyimide plastic.

A plurality of optically reflective discs 106, generally similar todiscs 54, are attached, typically by cementing, to an upper 2D surface110 of substrate 102. Discs 106, also referred to herein as reflectors106, are formed in a generally rectangular 2D pattern on surface 110.Discs 106 are distributed so that when illuminated and imaged by camera42 they are easily distinguished from substrate 102.

In addition, discs 106 are distributed with respect to an xz plane 120and a yz plane 122 through origin 103. xz plane 120 and yz plane 122 areplanes of asymmetry. Thus, discs 106 are arranged non-symmetrically withrespect to xz plane 120, so that the distribution of the discs on oneside of plane 120 do not mirror (through the plane) the discs on theopposing side of the plane. In addition, discs 106 are arrangednon-symmetrically with respect to yz plane 122, so that the distributionof the discs on one side of plane 122 do not mirror the discs on theopposing side of the plane.

In FIG. 4 discs 106 are shown as being distributed on sides of arectangle, however, it will be understood that this is but one examplefor the positioning of the discs on surface 110. Other distributions ofdiscs 106, providing that they define planes of asymmetry as describedabove, are also assumed to be comprised within the scope of the presentinvention.

Furthermore, it will be appreciated that the physical construction ofpatient marker 38 described above is by way of example. Thus,embodiments of the present invention comprise any patient marker formedof any conveniently shaped solid opaque substrate to which is attachedan optical pattern, the pattern defining planes of asymmetry asdescribed above.

FIG. 5 is a flowchart of steps performed to register patient marker 38with the anatomy of patient 22 during the initial preparatory stage of amedical procedure illustrated in FIG. 1 , according to an embodiment ofthe present invention. While the following description assumes, forsimplicity, a CT scan, other types of fluoroscopic imaging are alsoconsidered to be within the scope of the present invention.

In an initial step 150, medical professional 26 makes an incision in theback of patient 22, inserts spinous clamp 30 into the patient, and thenclamps the clamp to one or more of the processes of the patient.

In a patient marker step 152, the medical professional attaches patientmarker 38 to spinous clamp 30, ensuring that the marker is rigidlyattached to the clamp. Marker 38 is attached to clamp 30 so that surface110, corresponding to the xy plane of the xyz axes, is approximatelyparallel to a frontal plane of patient 22, xz plane of asymmetry 120 isapproximately parallel to a sagittal plane of the patient, and so thatyz plane of asymmetry 122 is approximately parallel to an axial plane ofthe patient. As used herein, the term “approximately parallel” asapplied to two planes indicates that the planes subtend an angle withina range of ±20° to each other.

In a registration marker step 154, the professional places registrationmarker 40 on the skin of the back of the patient, typically as close tothe patient's spine as is convenient.

In a camera step 156, professional 26 adjusts his/her position so thatcamera 42, attached to head-mounted display 64 images the registrationmarker and the patient marker. Professional 26 adjusts their position sothat the images formed by camera 42 of the registration marker and ofthe patient marker are clear images, i.e., that neither marker occludesthe other. Typically processor 32 of processing system 28 is configuredto verify the acceptability of the two marker images, and if necessarythe professional may use and communicate with system 28 to adjust, in aniterative manner, their position and/or that of the registration markeruntil system 28 provides an indication to the professional thatacceptable images are being generated.

Once acceptable images are being generated, a camera image of the twomarkers is acquired, and is provided to processing system 28.

In a fluoroscopic scan step 158, a CT scan of patient 22, in thevicinity of marker 40 is performed, and processing system 28 acquiresthe scan. The scan may be performed by inserting patient 22 into a CTscanning system so that marker 40 is scanned. The insertion may beimplemented by bringing the CT scanning system to patient 22, or bytransporting the patient to the system. In either case, marker 40remains in the marker's position of step 156.

In a scan analysis step 160, processor 32 analysis the CT scan acquiredin step 158, the scan comprising an image of radiopaque elements 60 andof the anatomy of patient 22. From the acquired image, processor 32calculates the position and orientation of registration marker frame ofreference 50, and registers the frame of reference with the anatomy ofthe patient. The registration typically comprises a set of vectors Pbetween selected points on registration marker 40 and selected vertebraeof patient 22. In one embodiment, the registration comprises using a 4×4homogenous transformation, comprising a 3×3 rotation and a 1×3translation, that transforms a point in the space of patient 22 to apoint in registration marker frame of reference 50.

In a camera image analysis step 162, processor 32 analyzes the cameraimage of patient marker 38 and registration marker 40 acquired in step156. From the acquired image, processor 32 calculates the position andorientation of registration marker frame of reference 50, and theposition and orientation of patient marker frame of reference 100. Oncethe processor has calculated the positions and orientations of the twoframes of reference, it formulates a registration of the two frames ofreference as a set of vectors Q describing the transformation of theregistration marker frame of reference to the patient marker frame ofreference.

In a concluding analysis step 164, the processor adds the two sets ofvectors found in steps 160 and 162 to formulate a registration set ofvectors R between the patient marker frame of reference 36 and thepatient anatomy, as shown in equation (1):R=P+Q  (1)

FIG. 6 illustrates a subsequent stage of the medical procedure, FIG. 7is a flowchart of steps performed during the subsequent stage, and FIG.8 shows schematic figures illustrating images generated in thesubsequent stage, according to an embodiment of the present invention.In the subsequent stage registration marker 40 has been removed from theback of patient 22, and medical professional 26 operates on the patientusing a surgical tool 190. The tool is tracked by the HMD processor, byhaving identifying reflectors 194, generally similar to reflectors 106,attached to the tool.

In an initial step 200 of the flowchart of FIG. 7 , the HMD projectsvisible or invisible light to patient marker 38 and tool 190. Camera 42acquires images of reflectors 106 of the marker, of reflectors 194 oftool 190 and of patient 22 and tool 190.

The flowchart then branches into two paths, a first path 202 and asecond path 204. Processor 32 implements steps of both pathssubstantially simultaneously.

In first path 202, in a three-dimensional (3D) image retrieval step 210,processor 32 retrieves a 3D stored patient anatomy image of patient 22,typically comprising a CT image of the patient, from stored images 35.The processor also retrieves a stored virtual image, also herein termeda stored representation, of tool 190 from the stored images.

In a 3D image presentation step 214, the processor presents aligned 3Dimages of the patient anatomy and of the virtual tool image in the headmounted display.

The position of the virtual tool image is determined from reflectors194. In order to ensure that the anatomy image and the virtual toolimage, projected by the display, align with the anatomy of patient 22and with the actual tool image, the processor determines the positionand orientation of frame of reference 100 of the patient marker from theacquired images of reflectors 106. The processor applies theregistration set of vectors R, found in step 164 of the flowchart ofFIG. 5 , to the position and orientation of the marker frame ofreference, so as to effect the alignment.

In second path 204, in a plane identification step 220, processor 32analyzes the images of reflectors 106 acquired by camera 42 to identifythe position and orientation of xz plane of asymmetry 120 and yz planeof asymmetry 122. From the images the processor also calculates andstores the height of camera 42 above the xy plane.

From the identified positions and orientations of the planes theprocessor determines on which side of the planes camera 42 resides. Eachplane has two sides, and it will be understood that the two planesdivide the volume around marker 38 into four regions, the cameraresiding in one of four regions.

In a tool reflector step 224 the processor analyzes the images ofreflectors 194 to find the position and orientation of tool 190.

In an image retrieval step 228 the processor retrieves a stored virtualimage of the tool. The processor also retrieves, from the stored 2Dimages, images of the patient anatomy at the tool position, and parallelto the axial and sagittal planes of the patient.

In an image presentation step 232, the processor uses the retrievedimages to generate a combined image of the patient anatomy with arepresentation of the tool superimposed on the patient anatomy, from apoint of view of the camera, i.e., from a point of view in the planesides identified in step 220.

The processor presents the combined image in HMD 64 for viewing byprofessional 26.

By presenting images in HMD 64 according to the point of view of camera42, embodiments of the present invention present correctly orientedimages to operator 26, who is wearing the HMD. It will also beunderstood that the correct orientation is determined according to theposition of the operator 26 with respect to the patient, i.e., whetherthe operator is to the left or right of the patient, and whether theoperator is on a lower or upper side of the patient.

FIG. 8 shows schematic illustrations of images generated in step 232,according to an embodiment of the present invention.

A diagram 300 illustrates an image 304A of tool 190 superimposed on animage 308A of the patient anatomy, from a point of view in a left sideof a sagittal plane of patient 22, and a diagram 312 illustrates animage 304B of tool 190 superimposed on an image 308B of the patientanatomy, from a point of view in a right side of the patient sagittalplane. The two diagrams are mirror images of each other, and use astored image 304 of tool 190. The two diagrams also use a stored image308 of the patient anatomy that is parallel to the patient sagittalplane at an identified position of tool 190.

A diagram 320 illustrates an image 304C of tool 190 superimposed on animage 324A of the patient anatomy, from a point of view in a lower sideof an axial plane of patient 22, and a diagram 330 illustrates an image304D of tool 190 superimposed on an image 324B of the patient anatomy,from a point of view in an upper side of the patient axial plane. As fordiagrams 300, 312, the two diagrams 320, 330 are mirror images of eachother, and use stored image 304 of tool 190. Diagrams 320, 330 use astored image 324 of the patient anatomy that is parallel to the patientaxial plane at the identified position of tool 190.

Returning to the flowchart of FIG. 7 , it will be appreciated thatprofessional 26 may select which images, referred to in steps 214 and232, are rendered for viewing in the head-mounted display. Thus theprofessional may view either the 3D images of step 214, or the 2D imagesof step 232, or both images simultaneously.

FIG. 9 is a schematic top-down view of surface 110 of marker 38, showingthe x, y, and z axes of the marker, as well as xz plane 120 and yz plane122.

As operator 26 moves from one side of xz plane 120 to the other side,then following on from step 232 of the flowchart of FIG. 7 together withthe diagrams of FIG. 8 , the images presented to the operator are mirrorimages of each other. The mirroring is also true when the operator movesfrom one side of yz plane 122 to the other side.

A disclosed embodiment of the present invention places a limitation onthe mirroring described above when moving from one side of a plane toanother, in order to reduce jitter in the presented images when theoperator is close to the plane. In order to reduce jitter, the processorconstructs transition regions around xz plane 120 and other transitionregions around yz plane 122. The following description is for thetransition region around xz plane 120 and to the right of yz plane 122.

Processor 32 constructs a first plane 402 containing and terminating atthe z axis, and at an angle +θ from xz plane 120, and a second plane 404containing and terminating at the z axis, and at −θ from xz plane 120.In one embodiment θ≤10°. The two planes form respective wedge-shapedregions 412, 414 with xz plane 120, and these two wedge-shaped regionscomprise the transition region around xz plane 120 and to the right ofyz plane 122.

If the movement across xz plane 120 includes both wedge-shaped regionsbeing crossed, by the HMD and the attached camera of the operator, orbegins from within one of the wedge-shaped regions and crosses the otherone, then the mirroring as described above is implemented.

However, if the movement across the xz plane does not comply with themovements above, e.g., the movement only crosses one wedge-shaped regionand stops in the other region, or only moves between wedge-shapedregions, then no mirroring is implemented.

For a transition region around xz plane 120 and to the left of yz plane122, the processor constructs two planes making angles γθ with the xzplane, generally similar to planes 402 and 404, so as to form two morewedge-shaped regions terminating at the z axis and to the left of the yzplane.

The processor constructs the same type of transition regions for yzplane 122. Thus, for a transition region around yz plane 122 and abovexz plane 120, the processor constructs two planes making angles γθ withthe yz plane, generally similar to planes 402 and 404, so as to form twowedge-shaped regions terminating at the z axis and above the xz plane.

Similarly, for a transition region around yz plane 122 and below the xzplane, the processor constructs two planes making angles ±θ with the yzplane, generally similar to planes 402 and 404, so as to form twowedge-shaped regions terminating at the z axis and below the xz plane.

There are thus a total of four transition regions distributedsymmetrically about the z-axis, each transition region comprising twowedge-shaped regions.

As for the movement for the illustrated transition region, if movementacross either of planes 120 or 122 includes both wedge-shaped regionsbeing crossed, by the HMD and the attached camera of the operator, orbegins from within one of the wedge-shaped regions and crosses the otherone, then the mirroring is implemented.

However, if the movement across either of the planes does not complywith the movements above, then no mirroring is implemented, i.e.,mirroring is precluded.

Another disclosed embodiment of the present invention places anotherlimitation on the mirroring described above. In this embodiment, whenthe operator moves to look over patient 22, mirroring is also precluded.To preclude mirroring for this embodiment, the processor checks if thecamera height, measured in step 220 of the flowchart of FIG. 7 haschanged, as is the case if operator 26 moves her/his head to look overpatient 22. I.e., if the camera height changes, no mirroring isimplemented regardless of whether the xz plane or the yz plane have beencrossed.

FIG. 10 is a schematic illustration of the subsequent stage of theprocedure, when two operators use an imaging system 320, according to anembodiment of the present invention. Apart from the differencesdescribed below, the operation of system 320 is generally similar tothat of system 20 (FIGS. 1-9 ), and elements indicated by the samereference numerals in both systems 20 and 320 are generally similar inconstruction and in function.

In contrast to system 20, system 320 is used by operator 26 and a secondoperator 326. Second operator 326 wears an HMD 364, and a camera 342 isfixedly attached to the HMD. HMD 364 and camera 342 are respectivelysubstantially similar in construction and function to HMD 64 and camera42. However, camera 342 is typically not used to perform theregistration described in the flowchart of FIG. 5 , since this isprovided by camera 42.

Images generated in HMD 364 are substantially as described in theflowchart of FIG. 7 . Thus, images presented in HMD 364 are orientedaccording to the point of view of camera 342, i.e., according to whetheroperator 326 is to the left or right of patient 22, and according towhether the operator is on the lower or upper side of the patient.

It will be understood that by presenting images in a head-mounteddisplay according to the point of view of the camera attached to thedisplay, embodiments of the present invention present correctly orientedimages to a wearer of the head-mounted display. It will also beunderstood that the correct orientation is determined according to theposition of the wearer of the HMD with respect to the patient, i.e.,whether the wearer is to the left or right of the patient, and whetherthe wearer is on a lower or upper side of the patient.

It will be further understood that for cases where there is more thanone HMD, each being worn by a respective wearer, embodiments of thepresent invention operate simultaneously and independently to presentcorrectly oriented images to each wearer, according to the position ofthe respective wearer with respect to the patient. A wearer on the rightside of the patient and a wearer on the left side of the patient arepresented with mirror images based on anatomy images parallel to thepatient sagittal plane; similarly a wearer on the lower side of thepatient and a wearer on the upper side of the patient are presented withmirror images based on anatomy images parallel to the patient axialplane.

It will thus be appreciated that the embodiments described above arecited by way of example, and that the present invention is not limitedto what has been particularly shown and described hereinabove. Rather,the scope of the present invention includes both combinations andsubcombinations of the various features described hereinabove, as wellas variations and modifications thereof which would occur to personsskilled in the art upon reading the foregoing description and which arenot disclosed in the prior art.

The invention claimed is:
 1. A computer-implemented method, comprising:storing in a memory a graphical representation of a tool used in aprocedure performed by an operator of an imaging system on a humansubject, and an image of anatomy of the human subject; acquiring imagesof a patient marker attached to the human subject and of the tool,wherein the images are acquired by a camera attached to a head-mounteddisplay of the imaging system configured to be worn by the operatorwhile the operator operates on the human subject using the tool;analyzing one or more images of the acquired images of the patientmarker and the tool, so as to identify whether the camera is located ona first side or on a second, opposing side of a sagittal plane of thehuman subject, or whether the camera is located on a first side or on asecond, opposing, side of an axial plane of the human subject; andrendering to the head-mounted display the image of the anatomy of thehuman subject with the graphical representation of the tool superimposedthereon from a point of view in the identified side of the humansubject.
 2. The method according to claim 1, wherein the analyzing ofthe one or more acquired images comprises identifying if the cameramoves from a first side to a second, opposing, side of the sagittal oraxial plane of the human subject, and if the camera moves from the firstside to the opposing side of the sagittal or axial plane of the humansubject, the rendering of the image of the anatomy of the human subjectwith the graphical representation of the tool superimposed thereoncomprises mirroring the image of the anatomy of the human subject withthe graphical representation of the tool superimposed thereon withrespect to the sagittal or axial plane, correspondingly.
 3. The methodaccording to claim 1, wherein the image of anatomy of the human subjectis a two-dimensional image.
 4. The method according to claim 1, whereinthe tool comprises identifying tool-reflectors attached thereto, andwherein the patient marker comprises optically reflective elementsdisposed on the patient marker so as to define at least one plane ofasymmetry thereon, by being disposed on opposing sides of the at leastone plane of asymmetry in a non-symmetrical arrangement with respect tothe at least one plane of asymmetry, so that the elements on a firstside of the at least one plane of asymmetry do not mirror, through theat least one plane of asymmetry, the elements on a second, opposing,side of the at least one plane of asymmetry.
 5. The method according toclaim 4, wherein the defined at least one plane of asymmetry of thepatient marker is parallel to the sagittal or axial plane of the humansubject, and wherein analyzing the acquired images comprises identifyingwhether the camera is located on a first side or on a second, opposingside of the defined at least one plane of asymmetry, and the renderingcomprises rendering the image of the anatomy of the human subject withthe graphical representation of the tool superimposed thereon from apoint of view in the identified side of the defined at least one planeof asymmetry.
 6. The method according to claim 4, wherein the at leastone plane of asymmetry makes an angle between +20° and −20° with thesagittal or axial plane of the human subject, correspondingly.
 7. Themethod according to claim 4, wherein the patient marker defines afurther plane and wherein the optically reflective elements are disposedon opposing sides of the further plane in a non-symmetrical arrangementwith respect to the further plane, further comprising analyzing the oneor more images of the acquired images so as to identify the furtherplane and to identify a side of the further plane wherein the camera islocated, and to render to the display the image of the anatomy of thehuman subject with the graphical representation of the tool superimposedthereon from a point of view in the identified side of the furtherplane.
 8. The method according to claim 1, wherein the camera is locatedat a vertical height above the patient marker, the method furthercomprising: ascertaining the vertical height in response to acquiringimages of the patient marker and the tool; calculating a pair of planes,each of the pair having a preset acute angle to one of the sagittalplane or the axial plane of the human subject and defining a firstacute-angled wedge region and a second acute-angled wedge region to theone of the planes; and when the display moves so that the point of viewcrosses the first acute-angled wedge region and the second acute-angledwedge region, or begins within the first acute-angled wedge region andcrosses the second acute-angled wedge region, while the camera remainsat the vertical height, rendering to the display the image of theanatomy of the human subject with the graphical representation of thetool superimposed thereon from a point of view in a side of the humansubject opposite the identified side of the human subject.
 9. The methodaccording to claim 1, wherein the camera is located at a vertical heightabove the patient marker, the method further comprising: ascertainingthe vertical height in response to acquiring images of the patientmarker; and when the display moves so that the vertical height changes,rendering unchanged to the head-mounted display the image of the anatomyof the human subject with the graphical representation of the toolsuperimposed thereon.
 10. The method according to claim 1, furthercomprising: analyzing additional images of the patient marker and thetool acquired by an additional camera, so as to identify whether theadditional camera is located on a first side or on a second, opposingside, of the sagittal or axial plane of the human subject, wherein theadditional camera is attached to an additional head-mounted display ofthe imaging system configured to be worn by an additional operator ofthe imaging system; and render to the additional head-mounted displaythe image of the anatomy of the human subject with the graphicalrepresentation of the tool superimposed thereon from an additional pointof view in the side of the human subject identified as the side in whichthe additional camera is located.
 11. The method according to claim 10,wherein when the operator is located on a first side of the sagittal oraxial plane of the human subject and the additional operator is locatedon the second, opposing side of the sagittal or axial plane of the humansubject, correspondingly, the rendering to the additional head-mounteddisplay of the image of the anatomy of the human subject with thegraphical representation of the tool superimposed thereon comprisesmirroring the image of the anatomy of the human subject with thegraphical representation of the tool superimposed thereon rendered tothe head-mounted display with respect to the sagittal or axial planecorrespondingly.
 12. An imaging system, comprising: a head-mounteddisplay configured to be worn by an operator of the system; a memoryconfigured to store a graphical representation of a tool used in aprocedure performed by the operator on the human subject, and an imageof anatomy of the human subject; a camera attached to the head-mounteddisplay and configured to acquire images of a patient marker attached tothe human subject and of the tool, while the operator operates on thehuman subject using the tool; and at least one processor configured to:analyze one or more images of the acquired images of the patient markerand the tool, so as to identify whether the camera is located on a firstside or on a second, opposing side of a sagittal plane of the humansubject, or whether the camera is located on a first side or on asecond, opposing, side of an axial plane of the human subject; andrender to the head-mounted display the image of the anatomy of the humansubject with the graphical representation of the tool superimposedthereon from a point of view in the identified side of the humansubject.
 13. The imaging system according to claim 12, wherein theanalyzing of the one or more acquired images comprises identifying ifthe camera moves from a first side to a second, opposing, side of thesagittal or axial plane of the human subject, and if the camera movesfrom the first side to the opposing side of the sagittal or axial planeof the human subject, the rendering of the image of the anatomy of thehuman subject with the graphical representation of the tool superimposedthereon comprises mirroring the image of the anatomy of the humansubject with the graphical representation of the tool superimposedthereon with respect to the sagittal or axial plane, correspondingly.14. The imaging system according to claim 12, wherein the image ofanatomy of the human subject is a two-dimensional image.
 15. The imagingsystem according to claim 12, wherein the tool comprises identifyingtool-reflectors attached thereto, and wherein the patient markercomprises optically reflective elements disposed on the patient markerso as to define at least one plane of asymmetry thereon, by beingdisposed on opposing sides of the at least one plane of asymmetry in anon-symmetrical arrangement with respect to the at least one plane ofasymmetry, so that the elements on a first side of the at least oneplane of asymmetry do not mirror, through the at least one plane ofasymmetry, the elements on a second, opposing, side of the at least oneplane of asymmetry.
 16. The imaging system according to claim 15,wherein the defined at least one plane of asymmetry of the patientmarker is parallel to the sagittal or axial plane of the human subject,and wherein the analyzing of the acquired images comprises identifyingwhether the camera is located on a first side or on a second, opposingside of the defined at least one plane of asymmetry, and the renderingcomprises rendering the image of the anatomy of the human subject withthe graphical representation of the tool superimposed thereon from apoint of view in the identified side of the defined at least one planeof asymmetry.
 17. The imaging system according to claim 12, wherein thecamera is located at a vertical height above the patient marker, and theprocessor is further configured: to ascertain the vertical height inresponse to the acquired images of the patient marker and the tool; tocalculate a pair of planes, each of the pair having a preset acute angleto one of the sagittal plane or the axial plane of the human subject anddefining a first acute-angled wedge region and a second acute-angledwedge region to the one of the planes; and when the display moves sothat the point of view crosses the first acute-angled wedge region andthe second acute-angled wedge region, or begins within the firstacute-angled wedge region and crosses the second acute-angled wedgeregion, while the camera remains at the vertical height, to render tothe display the image of the anatomy of the human subject with thegraphical representation of the tool superimposed thereon from the pointof view in a side of the human subject opposite the identified side ofthe human subject.
 18. The imaging system according to claim 12, whereinthe camera is located at a vertical height above the patient marker, andthe processor is configured: to ascertain the vertical height inresponse to the acquired images of the patient marker; and when thehead-mounted display moves so that the vertical height changes, torender unchanged to the head-mounted display the image of the anatomy ofthe human subject with the graphical representation of the toolsuperimposed thereon.
 19. The imaging system according to claim 12,further comprising: an additional head-mounted display configured to beworn by an additional operator of the imaging system; an additionalcamera attached to the additional head-mounted display and configured toacquire additional images of the patient marker and the tool, whereinthe processor is further configured to: analyze the additional images ofthe patient marker and the tool, so as to identify whether theadditional camera is located on a first side or on a second, opposingside, of the sagittal or axial plane of the human subject; and render tothe additional head-mounted display the image of the anatomy of thehuman subject with the graphical representation of the tool superimposedthereon from an additional point of view in the side of the humansubject identified as the side in which the additional camera islocated.
 20. The imaging system according to claim 19, wherein when theoperator is located on a first side of the sagittal or axial plane ofthe human subject and the additional operator is located on the second,opposing side of the sagittal or axial plane of the human subject,correspondingly, the rendering to the additional head-mounted display ofthe image of the anatomy of the human subject with the graphicalrepresentation of the tool superimposed thereon comprises mirroring theimage of the anatomy of the human subject with the graphicalrepresentation of the tool superimposed thereon rendered to thehead-mounted display with respect to the sagittal or axial plane,correspondingly.